Skip to content

feat(promptfoo): validate exported provider configs#1709

Merged
christso merged 2 commits into
mainfrom
feat/av-lbcv-promptfoo-export-validation
Jul 7, 2026
Merged

feat(promptfoo): validate exported provider configs#1709
christso merged 2 commits into
mainfrom
feat/av-lbcv-promptfoo-export-validation

Conversation

@christso

@christso christso commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add a focused scripts/export-promptfoo-config.ts path that exports representative AgentV eval YAML into a Promptfoo config for validation.
  • Preserve Promptfoo-native colon provider IDs and labels, lower agentv:codex-cli to a generated Promptfoo-loadable TypeScript custom provider module, and keep validation free of live provider calls.
  • Export the supported host/filesystem environment subset into a generated Promptfoo beforeAll extension, while passing the resolved isolated workdir through provider config, default-test vars, and metadata.
  • Fail Docker-shaped environment export with a clear diagnostic instead of degrading isolation, image/context, mounts, services, resources, secrets, or provenance.
  • Add a CI Promptfoo Export job that runs exporter tests, exports the host environment fixture, and validates it with promptfoo validate config.
  • Document direct Promptfoo provider compatibility vs AgentV-only compatibility by export.

Dependency Notes

This PR is stacked on feat/av-uttb-provider-surface-migration-docs because it builds on that provider-surface docs/migration branch. I did not merge feat/av-ctfu-providers-schema: that active worktree currently has uncommitted parser/schema changes owned by worker av-ctfu-providers-schema, so this exporter stays out of those core parser files.

Promptfoo 0.121.15 and 0.121.17 both reject provider IDs with a file://...ts:callApi suffix during config validation. The exporter therefore emits the accepted TypeScript custom provider module form file://.agentv/generated/promptfoo/providers/codex-cli-provider.ts while the generated module also exposes callApi internally.

Validation

  • bun run validate:promptfoo-export
  • bunx biome check scripts/export-promptfoo-config.ts scripts/export-promptfoo-config.test.ts scripts/fixtures/promptfoo-export/host-environment.agentv.yaml scripts/fixtures/promptfoo-export/docker-environment-unsupported.agentv.yaml .github/workflows/ci.yml apps/web/src/content/docs/docs/next/reference/promptfoo-parity.mdx
  • tmpdir=$(mktemp -d) && bun scripts/export-promptfoo-config.ts --input scripts/fixtures/promptfoo-export/host-environment.agentv.yaml --output "$tmpdir/promptfooconfig.yaml" && bunx promptfoo@0.121.15 validate config -c "$tmpdir/promptfooconfig.yaml"
  • bun run typecheck
  • git diff --check

Live dogfood was not run because this change only exports and validates/config-loads Promptfoo config and does not execute AgentV eval/provider/grader behavior.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 7, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 947489b
Status: ✅  Deploy successful!
Preview URL: https://33340354.agentv.pages.dev
Branch Preview URL: https://feat-av-lbcv-promptfoo-expor.agentv.pages.dev

View logs

@christso christso force-pushed the feat/av-uttb-provider-surface-migration-docs branch from 1438ab3 to 7b74d65 Compare July 7, 2026 02:16
Base automatically changed from feat/av-uttb-provider-surface-migration-docs to main July 7, 2026 02:30
@christso christso force-pushed the feat/av-lbcv-promptfoo-export-validation branch from 639d396 to 947489b Compare July 7, 2026 02:37
@christso christso marked this pull request as ready for review July 7, 2026 02:38
@christso christso merged commit a688de1 into main Jul 7, 2026
11 checks passed
@christso christso deleted the feat/av-lbcv-promptfoo-export-validation branch July 7, 2026 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant